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Abstract 



This article reports the results of a survey that evaluates the state of 
assessment in Maryland* Ancillary objectives of this study were: (1) to 
identify the cognitive or noncognitive areas to be assessed, (2) to investigate 
perceptions about the role of the institutional researcher in assessment 
activities and, (3) to solicit information that would guide the formation of a 
consortium focused on assessment A proposed legislative mandate, lacking 
external fimding, was not found to be sufficient impetus for the initiation of 
assessment programs* Institutions were more compelled to respond to 
requests from external agengies, Insitutional researchers were found to play 
a key role in the assessment process* The majority of respondents were 
interested in a consortium based on institution type. 
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The State of Assessment in Maryland: 
Responses from Post Secondaiy Instil 

In 1984, a survey of higher education institutions listed several deterrents 
to assessment. Many (71%) of the responding institutions reported having no 
funds to develop procedures while 64% were not clear about what to evaluate 
and 60% expressed some fear about the misuse of results (El-Khawas, 1985). 
However, three years later, Boyer, Ewell, Finney and Mingle (1987) reported 
that approximately 34 states had developed "formal" requests for assessment 
for their public institutions. By 1989, El-Khawas (1989) described a national 
picture of assessment using data collected from 366 postsecondary institutions. 
Close to 70% of responding institutions reported ongoing assessment activities 
with about 55% developing instruments for student-oriented assessment. 
There was a seven percent increase (34%) in the nximber of institutions which 
reported collaborating with other institutions on their assessment activities 
since 1988. To further illustrate the momentum of the assessment movement, 
the 1990 American Cotmcil on Education's national survey results indicated 
that 80% of the responding institutions had on-going assessment activities 
(Blumenstyk & Magner, 1990). 

In many states, an institution's initiative to begin an assessment 
program has been not only tied to mandates, but also to the availability of funds 
specifically allocated for the development of assessment projects. El-Khawas 
(1989) reported that 40% of the total responding institutions admitted that their 
assessment efforts were developed in response to drives from external agencies 
such as state boards or accrediting bodies. Only 30% of the institutions 
surveyed were developing assessment strategies without external mandates, 
lastitutions in Colorado faced a loss of as much as two percent of their budgets 
if assessment programs were not fully developed by 1990. The prospect of 
deficit funding caused Metropolitan State to consider charging students to 
recoup assessment expenses. 

Without such charges to students, mandated assessment programs 
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might cause administrators to reallocate or divert fimds from other campus 
programs. Certain states, such as New Jersey, Tennessee and Virginia, have 
practiced incentive funding to prohibit such a shifting of funds. Institutions in 
Tennessee have been eligible for incentive funds since 1979 if they could 
demonstrate quality. In 1987, the Tennessee Higher Education Conmiission 
established new guidelines which required students to take specific 
standardized tests. The scores of these students had to improve if additional 
funds were to be awarded. In contrast. South Dakota had also developed 
test-centered assessment procedures which were tabled in 1987 due to a lack of 
funds (The Chronicles of Higher Education, a & b, 1988), 

Maryland institutions of higher education faced a similar situation to 
that experienced by South Dakota, In 1986, the Maryland State Board for 
Higher Education developed a plan to test sophomores in public two and four 
year institutions. The test would measure higher order general intellectual 
skills and/or the general education component of students* programs. The 
results of a pilot program were to determine if a statewide approach to testing 
should continue or whether tests should be tailored to institutional 
characteristics. This effort was tabled when the state did not fund the pilot 
program (State Board for Higher Education, 1986), 

Clearly, the allocation of funds specifically earmarked for assessment 
and/or a state mandate requiring assessment provide strong incentives for the 
development of assessment programs. This fact raises the question of the 
types of assessment activity that have developed in states where neither funds 
nor state mandates about assessment had been issued. 

Inasmuch as there had been no information collected at the state level 
that detailed assessment in Maryland, the primary purpose of this study was 
to describe the state of assessment in Maryland, Ancillary objectives of this 
study, developed to answer some questions of members of a Maryland 
Association of Institutional Research assessment group were: (1) to identify 
the cognitive or noncognitive areas assessed, (2) to investigate perceptions 
about the role of the institutional researcher in assessment activities and, (3) 
to solicit information that would guide the formation of a consortium focused 
on assessment. 
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Saupe (1990) suggested that the responsibilities of institutional research 
offices varied from planning and budgeting functions to assisting in the 
improvement of teaching and learning, Rogers and Gentemann (1989) 
evaluated the role of institutional research in a 1987 study of 167 institutions 
that would receive reaccreditation from the Southern Association of Colleges 
and Schools between 1987 and 1992, Rogers and Gentemann found institutional 
research tasks in these institutions to be most related with assessment 
activities. However, numbers of institutions such as James Madison 
University, Western Michigan, Towson State University and Goucher College 
have established offices of assessment which are independent of Institutional 
Research offices. Given this fact, the role of the institutional research office in 
developing strategies for assessment was targeted for further study. 

Rogers and Gentemann also determined that only 44% of the respondents 
had defined student outcomes on their campuses. Cry and Parker (1989) 
reported similar results from a telephone survey of 33 xmiversities. Less than 
20% reported assessing the gains in academic skills or in the major area as a 
result of the university experience. Most institutions that reported having 
some assessment activities mentioned mostly entry-leve^ placement tests, 
ratings of instructors and courses, and alimmi surveys. Therefore, though 
high proportions (80%) of institutions reported assessment activities to improve 
teaching and learning, there were few data on student outcomes to validate 
this claim. However, in 1990, the American Council on Education foimd that 
94% of institutions with assessment programs were evaluating basic skills, 
67% were assessing general education and liberal studies, and 62% were 
measuring students' progress in their majors (Blumenstyk & Magner, 1990). 

This paper intends to serve as a case study of the types of assessments 
that occur on postsecondary campuses where state mandates are planned but 
nonexistent, and also to describe the other factors which may impede or drive 
assessment on various campuses. In addition, as the functions of institutional 
research offices may vary on many campuses, the information gathered by 
this study describes the roles institutional researchers may fill regarding 
assessment activities. 



ERLC 



6 



State of Assessment 
6 



Mietiiodolog}r 

In 1987, the Maryland Association of Institutional Research (MdAir) 
created an Outcomes Assessment Special Interest Group (SIG) to address 
assessment issues on area campuses* The SIG was to become the mediimi for 
achieving two important objectives: (1) creating a means for people from all 
segments of higher education to interact with one another, and (2) giving 
presentations and holding workshops during annual conferences. The 
assessment SIG began with only four members, but over the course of one year 
grew to approximately fifty members* The group was renamed the Maryland 
Assessment Consortium and new members were recruited from across the 
state. 

In the summer of 1989, the Consortium decided to survey all institutions 
in Maryland on assessment issues. The purposes of our survey were 
three-fold: (1) to identify assessment activities in the state, (2) to mobilize 
institutions to join the consortium, and (3) to clarify the role of institutional 
researchers regarding assessment. 

To construct the survey, members of the Consortium were asked to 
formulate survey questions appropriate for identifying assessment activities in 
Maryland. Approjdmately 25 fixed-format and open-ended questions were 
developed for the survey. In July, drafi;s of the survey were mailed to all 
members of the Consortium, who then edited the questions and provided 
additional suggestions. 

The final survey contained four sections. The first section asked 
respondents to describe the institution's characteristics: i.e., type of 
institution, and fiill- and part-time enrollment. The second section dealt with 
assessment issues on the campus, such as who initiated assessment on the 
campus, v/hy the assessment process was initiated, who was responsible for 
assessment, which students were assessed, what procedures were followed, 
and how the data were distributed and used on the campus. The third section 
asked respondents to indicate how far along they were in an assessment 
process within 31 identified areas. Respondents also could supply additional 
areas of assessment. The stages of assessment were described along a 
continuum that ranged from planning, defining, and measuring to 
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distributing results. Each of these stages was defined and one example was 
presented that described the assessment process at an imaginary institution. 
The final section dealt with consortium issues: (1) types of consortia of 
interest to the respondents, (2) the role MdAir should play in statewide 
assessment, and (3) types of activities of interest to respondents. 

In September of 1989, 57 surveys were mailed to the president's offices of 
all institutions of higher education in Maryland, Each survey was 
accompanied by a self-addressed stamped envelope and a cover letter. The 
cover letter requested that the president pass the sur/ey on to the person on 
campus currently responsible for learning/outcomes assessment. Institutions 
were assured of anonymity and promised a copy of the results, 

A month later, follow-up calls were made to institutions to determine 
whether or not the inGtitutions had responded and to encourage the 
institutions to reply if they had not already done so, A second survey was 
mailed when necessary. 

Of the 57 surveys mailed, 24 were returned, yielding an overall response rate 
of 42%, The greatest perceixtage of our respondents were from public two year 
institutions. Private and public comprehensive institutions accounted for 21 
and 17 percent of the responses, respectively. Graduate schools and 
imiversites each accounted for eight percent of the responses - the number of 
private and public graduate schools and universities was equally distributed. 
Institutions v/ere also asked to specify the number of full- and part-time 
students enrolled. Half of the responses were from institutions enrolUng fewer 
than 2000 full-time students. 

From our conversations with institutional representatives during 
follow-up phone calls, several reasons were identified to explain non-returns. 
The majority of non-returns were from larger institutions and state schools. 
Several institutions were in transition: they had a new president, a new 
institutional research director, or were currently creating an assessment 
office. Some institutional representatives were unable to locate an individual 
on their campus responsible for assessment, A few were unclear about the 
term "assessment". In addition, some institutional representatives said they 
were embarrassed to report that they did not yet have an assessment program. 
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Others simply did not view the survey as important: they filed it, or put it off 
past the deadline. 

Maryland Assessment Programs: Results £rom Area Institutions 

Development and Org anization of Asses s ment Programs 

The second section of the survey asked questions pertaining to the 
organization and development of assessment programs at area campuses. 
Respondents were asked if their institutions had, or were developing student 
assesSi^ent programs. The term "assessment" was not defined in the survey, 
allowing institutions to define "assessment" individually. Three-fourths of the 
resnondents indicated that their institutions either had or were developing a 
student assessment program. The remaining 25 percent of the respondents 
indicated that such programs were absent from their campuses. If 
respondents indicated that they did not have an assessment program, they 
were instructed to go to the next section of the survey. 

Respondents who indicated that they had developed or were developing a 
student assessment program were next asked how many years ineir student 
assessment plan had been in operation. The mean length of operation for 
student assessment plans was 4.8 years; the most firequent response, however, 
was three years. 

Respondents were asked to specify who initiated ^Jie student assessment 
process at their institution, why the process was initiated, who developed the 
institution's assessment program, and Vrho was responsible for implementing 
the assessment program. These questions were multiple response items in 
which respondents were asked to check all responses that applied. 

The campus units most firequently cited for intitiating the student 
assessment process were faculty (39%) and academic affairs (39%) (see Table 
1). Academic affairs was most frequently cited as ha:ving responsibility for 
implementing the assessment program (44%). Interestingly, because 
Maryland is under a state mandate to provide evidence of public accountability 
(Article-Education §11-304, 1989), only 11 percent of the respondents indicated 
that legislature was responsible for initiating student assessment 
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Ibsert Table 1 about hei^ 



When respondents were asked why the assessment process, was initiated, 
the most frequent response given was to address issues of accountabiUty (61%), 
and the second was to respond to accrediting agencies (44%) (see Table 2)* 
Campus committees or task forces were reported most often as the formulators 
of the student assessment program (71%) (see Table 3)* Eighteen percent of 
responding institutions indicated that the institutional research office was 
responsible for formulating student assessment programs* Eleven percent 
reported that the institutional research office was responsible for 
implementing student assessment programs (see Table 4)* However, 63% of 
the respondents indicated that an institutional researcher was involved in the 
assessment program development process primarily as a campus committee 
member (41%) and/or as an advisor (41%). 



Ihsext Tables 2^ 3, and 4 about here 

Respondents were asked to identify what type of student testing 
procediu-es were currently in effect at their institution* Seventy-one percent of 
the respondents indicated that they used mandatory testing; the majority of 
these respondents noted that they were referring to required placement 
testing* 

The final questions concerning campus assessment practices focused on 
what student populations the institutions planned to assess, how the 
assessment data were used on campus, to whom the assessment results were 
distributed and lastly, the impact of assessment results on budget priorities* 
A wide range of responses were given to identify the student cohorts the 
institution planned to assess* Coiiimon responses to this question included all 
entering freshmen and altmmi* 

Once assessment data were collected, the results were reportedly 
distributed to faculty (76%) and administration (71%), and were used primarily 
to improve teacliing and learning (83%) and to evaluate programs (78%). In 
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terms of the impact that assessment results have on budget priorities, 53% of 
the respondents indicated that assessment results did influence budget 
priorities, especially in terms of resource allocation and strategic planning. 
The Assessment of Cognitive and Noncognitive Outcomes 

Seventeen out of 24 (71%) institutions responded to the assessment 
process section of the survey. Thirty-one assessment areas were categorized 
according to whether they were cognitive or noncognitive measures. The 
cognitive subcategory was further subdivided into skills, ability, competency, 
and knowledge. The noncognitive subgroup included measures of attitudes 
and institutional efifectivi^ness. 

To determine the appropriate subgroup for an area, definitions were 
developed. Areas within the cognitive domain were defined as follows: (a) 
cognitive skills - intellectual processes of knowing; (h) ability - an inherent or 
learned proficiency; (c) competency — a developed or learned sufiRciency; and 
(d) knowledge — an awareness of information. The definitions of the 
noncognitive areas were: (a) attitude — a feeling towards a person or thing and 
(b) measures of institutional effectiveness — indicators which describe the 
degree to which an institution was successful in fulfilling its mission and 
goals. 

Cognitive Domain, Cognitive skills included the areas of writing, 
reading, critical thinking, speaking, quantitative skills and problem solving. 
Forty-four percent of responding institutions were in the planning stages 
assessing cognitive skills. Speaking skills appeared to be the least desired area 
to assess with 48% of the institutions reporting, "Not Applicable". The 
Academic Profile, alumni follow-up and employer surveys were the 
instruments most often used to assess cognitive skills. 

Respondents reported collecting data about general education, knowledge 
in the major, creativity, performance on national standardized tests, 
admission to graduate school, performance on licensing exams, and test 
scores for professional degrees. These indicators were categorized as 
measures of knowledge, competency, and ability. Most institutions (70% of 
those involved in assessment) were in the planning stages of measuring 
general education. Many institutions were distributing the results from 
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professional and technical programs* Instruments used included nursing, 
teaching, and other various medical licensing exams* 

Non-Cognitive Domain. The subcategory of attitudes included the areas 
of political views; campus environment; awareness of moral issmes and 
different cultures; different viewpoints; the collective appreciation of art, 
music, and drama; the appreciation of literature, science and technologjr, and 
socioeconomics* Compared to other subcategories, most institutions reported 
little interest in assessing moral issues, different cultures, political attitudes 
or tolerance of different viewpoints* Most institutions reported that they were 
distributing the results of a campus environment survey* 

There was mixed interest in areas of appreciation of art, music, drama, 
literature, and science and technology* Most two year institutions were not 
interested in assessing these areas while some four year insiititutions were 
planning, measuring or distributing the results of assessment about 
appreciation* 

Institutions reported udng several methods to measure institutional 
effectiveness; namely, (1) alumni surveys, (2) commxmity coUegie enrollment 
reports, (3) employer surveys, (4) institutional research reports, (5) registrars* 
records, (6) Maryland State Board for Community Colleges* reports, (7) 
Maryland Higher Education Commission's Enrollment Information System 
data (8) state reports (9) graduation rates, and (10) retention rates* Many of 
these reports were developed by or produced for institutions for external agency 
reporting* Few institutions ir'^icated assessments of admission to graduate 
schools or professional degrees* Both two and four year institutions reported 
that they were distributing results about institutional effectiveness* 

Eight additional areas of assessment were supplied by respondents: 
interaction with others, personal growth and well-being, functioning in the 
work environment, professional development, freshman characteristics, 
personal development, stage theory, and personality type* 

In summary, 71% of the responding institutions reported being in some 
stage of assessment* Most institutions were: (1) planning to measure cognitive 
skills and general education, (2) distributing results that described 
institutional effectiveness and professional and technical programs, and (3) 
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undecided about measuring liberal arts appreciation. 

Maryland Assessment C!onsorti um 
The purpose of the last section of the questionnaire was to obtain 
information that would assist the members of the Consortium in further 
defining the role of an assessment group in Maryland and the role of 
institutional researchers in the assessment process* 

Sixty-two percent of those who responded were interested in joining the 
Consortium and another 38% were not ready to join, but wanted to be kept 
informed. The majority (77%) of the respondents were interested in a 
consortium that was based on institution type, e.g., liberal arts, 
comprehensive, public or private two or four year institutions. Approximately 
one-third of the respondents were interested in discipline and/or skill-based 
consortia. 

Repondents were asked to rank, in order of priority, the appropriate roles 
of institutional researchers regarding statewide assessment. Table 5 
illustrates the rankings assigned to each role. 



Ltisert Table 5 about here 



One-half of the respondents reported sharing information and strategies 
about assessment as the number one priority of institutional researchers. 
Providing technical assistance about assessment was selected as a priority by 
32% of the respondents. Less than ten percent of those who responded 
indicated that institutional researchers should develop assessment policy. 

Over fifty percent of the respondents were interested in participating in 
assessment-related seminars, conferences, and Consortium meetings. The 
majority of respondents were also interested in receiving information, 
including a list of consultants, descriptions of assessment strategies, sample 
test instnmients and newsletters. 
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Discussion 

The 1988 legislation, Maryland Senate Bill 459, mandated that each 
public higher education institution in the state develop an accountability plan 
based on its individual mission. Senate Bill 459 required the accoimtability 
plan to include a clear statement of outcomes and an evaluation of those 
outcomes. Interestingly, only 11 percent of responding institutions indicated 
that their campus assessment programs were initiated in response to the 
legislation. It was also surprising that one year after the legislation was 
enacted, a quarter of the responding institutions reported that they were not yet 
in the development stages of establishing an assessment program. 

The proportion (42%) of responding institutions was lower than expected 
given the passage of Senate Bill 459 ^d may limit the generalizability of the 
results. One reason for nonresponse centered on misunderstandings about the 
meaning of the term, "assessment". "Assessment" intentionally was not 
defined in the survey to permit individual interpretation. Unfortunately, the 
absence of a formal definition may have lowered the overall response rate to 
the survey. Follow-up calls revealed that some institutions did not return 
surveys because individuals who initially received the survey were unfamiliar 
with the term "assessment" and were unsure of the appropriate offices to 
receive the survey. 

The question remains, did the lack of targeted funding result in an 
absence of outcomes assessment in Maryland? The lower response rate may 
suggest an answer: a legislative mandate without fimds or guidelines lacked 
the support required to ensure compliance by all public postsecondary 
institutions. On the otherhand, the survey's results do suggest that 
institutions have completed assessments of those areas required by external 
agencies. 

Results of the survey in the areas of assessment of cognitive and 
noncognitive domains suggest that external mandates have been critical to the 
implementation of assessment in specific areas. Institutions appear to be 
better prepared to respond to external agencies with completed reports than to 
address issues about which there were no requests for data. For example, 
survey respondents reported distributing the results of technical and 



ERLC 



14 



State of Assessment 
14 



professional programs, such as nursing, which are regulated by accrediting 
bodies. On the other hand, most institutions were in the planning stages 
regarding the measurement of cognitive skills. In addition, responding 
institutions were diss-^minating assessment results, such as graduation rates, 
to state boards and commissions. In all cases, two and four year institutions 
listed reports that were required by external agencies as measures of 
institutional effectiveness. 

The measurement efforts in noncognitive areas appeared to vary on the 
basis of institutional type and interest. Two year institutions were less likely to 
assess students' appreciation of art, music, and drama than responding four 
year institutions. Four year institutions with liberal arts based ciirricula may 
have more reason to focus attention on the measurement of cultural 
appreciation. Commimity colleges, on the other hand, may have greater 
interest in assessing technical and professional programs. 

For the most part, institutional researchers in Maryland have been 
responsible for collecting assessment data and for developing corresponding 
reports. However, as institutions begin to measure additional academic areas 
within the campus, academic administrators and faculty may assimie larger 
roles in the assessment process. Survey results support this hypothesis. 
While academic administrators and faculty were more likely to have been the 
initiators, implementors of, and recipients of the results of assessment 
programs, institutional researchers described their roles as advisors and/or 
providers of information. The survey results revealed that the primary uses of 
assessment data were for academic reasons: the improvement of learning, 
teaching, and program evaluation. Therefore, describing the institutional 
researchers* role in assessment as advisory rather than policy-making was 
both experiential as well as pragmatic. 

Fitting with their roles as advisors and providers of information, 
institutional researchers indicated a need to obtain assistance about the kind of 
advice and information they could provide for their institutions. The survey 
results described the kinds of activities critical to institutional researchers 
involved with assessment programs. 

In a state which provided few incentives to assesp, the establishment of a 
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statewide assessment consortia appealed to respondents. Institutional 
researchers suggested that consortia could provide: (1) a forum for sharing 
information, (2) meaningful and interesting assessment-related activities, 
such as seminars and conferences, and (3) a means of sharing the 
consequences of various forms of student outcomes measurement at similar 
institutions* In addition^ consortia should serve as clearinghouses for 
assessment-related material and consultants* 

In summary, the survey results served as a baseline measure of the 
assessment activities that occurred on some Maryland college campuses 
during 1989* If accountability reports are firmly mandated by the Higher 
Education Coordinating Commission in upcoming years, it may be worthwhile 
to replicate this study to determine how campuses respond to internalize the 
assessment process* 
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TABLE 1. 

Who initiated the student assessment process at vour intitution? 

(N=18) 



N % 



Academic Affairs 


7 


39% 


Faculty 


7 


39% 


Statewide Board 


4 


22% 


Trustees/Board 


4 


22% 


President 


4 


22% 


Student Affairs 


3 


17% 


Other 


3 


17% 


Legislature 


2 


11% 
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TABLE 2. 

Why was the student asspssmen.t process initiated? 

(multiple response) 
(N=18) 



N % 



To address issues of accountablitiy 


U 


61% 


To respond to accrediting agencies 


8 


44% 


To facilitate insitutional planning 


7 


39% 


To increase student success 


2 


11% 


To enhance student learning 


2 


11% 


To facilitate academic advising 


1 


6% 


To respond to academic reorganization 


1 


6% 


To better meet student needs 


1 


6% 



ERIC 



19 



State of Assessment 
19 



TABLE 3. 

Please identify the group or individual responsible for the formulation of vour 
student assessment program 

(N=17) 



N % 



Campus committee/Task Force 32 71% 

Academic Affairs 5 29% 

Institutional Research 3 18% 

Student Affairs 3 18% 

Faculty Committee 1 6% 

University Senate Self-Study Committee 1 6% 
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TABLE 4. 

Who is responsible for implementing the assessment program? 

(multiple response) 
(N=18) 



N % 



Academic Affairs 8 44% 

Campus committeeyTask Force 6 33% 

Student Affairs 4 22% 

Institutional Research 2 11% 

Assessment Office 2 11% 

President's Office 2 6% 

Faculty Senate 2 6% 
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TABLE 5. 

Ranked Asses s ment Roles for Institutional Researchers 

!• Share information and strategies about assessment 

2* Provide technical assistance in the area of assessment* 

3* Participate in a fonmi for assessment discussions or "like" 
institutions* 

4* Evaluate and share information concerning consultants* 
5* Consult with our membership* 
6* Develop assessment policy* 
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